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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

/ A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 



- Extensions of time may be available under the provisions of 37 CFR 1 . 1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

!)□ Responsive to communication(s) filed on 30 July 2003 . 
2a)D This action is FINAL. 2b)E3 This action is non-final. 

3) Q Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) M7 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) [X] Claim(s) 1-17 is/are rejected. 

7) Q Claim(s) is/are objected to. 

8) Q Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) Q The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) £3 Notice of References Cited (PTO-892) 
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U.S. Patent and Trademark Office 
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DETAILED ACTION 

1 . Claims 1 -1 7 are presented for examination. 

Drawings 

2. The Drawings filed on 7/30/2003 are acceptable for examination purpose. 

Priority 

3. Applicant's claim for domestic priority under 35 U.S.C. 1 19(e) is acknowledged 
based on the provisional application 60/400, 143 filed on 7/31/2002; 60/400,213, filed on 
7/31/2002.. 



Information Disclosure Statement 

4. The information disclosure statement filed on 1 1/28/2003 is in compliance with 
the provisions of 37 CFR 1.97, and has been considered and a copy is enclosed with 
this Office Action. 



Specification 

5. At page 27, para 0116, "Mem ry Blocking" should be " Memory Blocking " 
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Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S.C. 103(a). 

6. Claims 1-17 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Gall US Patent No. 6480862 published on November 12,2002 in view of, Richard S 
Sutton [hereafter Sutton], "Generalization in Reinforcement Learning: Successful 
Examples Using Sparse Coarse Coding", Advances in neural Information 
processing systems 8, 1996, pp 1038-1044. 

7. As to claim 1,12, Gall teaches a system which including 'memory management 1 
[col 2, line 28-37, col 4, line 50-60, col 6, line 50-56]; 'a computer system [see fig 2, 
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element 30] or virtual machine [fig 3, element 44] having a memory space [fig 2, 
element 32a, col 4, line 50-60]. It is however, noted that Gall does not specifically teach 
'reinforcement learning is used to control the management', although Gall specifically 
suggests virtual machine implementing Java Virtual Machine as detailed in fig 2, also 
teaches garbage collection related memory management algorithms [col 8, line 15-23]. 
On the other hand, Sutton disclosed Yeinforcement learning is used to control the 
management' see Abstract, page 1-2, specifically teaches "reinforcement learning 
applied for example in dynamic programming. 

It would have been obvious to one of the ordinary skill in the art at the time of 
applicant's invention to incorporate the teachings of Sutton into relation-based ordering 
of objects in an object heap specifically directed to ordering, accessing data objects and 
their relationships between multiple objects in virtual memory scheme of Gall because 
both Sutton, and Gall directed to temporal and dynamic programming, more 
specifically, Gall is directed to heap managed garbage collection in multiple computer 
programs executing simultaneously on the same computer system [Gall: col 8, line 36- 
42, fig 3], while Sutton is directed to improving approximation, estimation based on 
reinforcement learning specifically implementing algorithms models that executes the 
task dynamics and applied dynamic programming backups [Sutton: page 2]. 
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One of the ordinary skill in the art at the time of applicant's invention to 
incorporate the teachings of Sutton into relation-based ordering of objects in an object 
heap specifically directed to ordering, accessing data objects and their relationships 
between multiple objects in virtual memory scheme of Gall because that would have 
allowed users of Gall to use "reinforcement learning system" involving not only multiple 
step decision with respect to current actions, but also evaluates and actions under 
uncertainty or unprededictable actions because reinforcement learning algorithms 
supports state transitions improving overall performance as suggested [Sutton: page 2]. 

8. As to claim 2, 13, Gall disclosed 'management of the memory or storage space 
includes a garbage collection process' [fig 3, fig 8, col 13, line 1-4]. 

9. As to claim 3, 8, 14, Gall disclosed Virtual machine is a Java Virtual Machine' [fig 
3, element 44, col 7, line 60-63]. 

10. As to claim 4, 9, 15, Sutton disclosed 'reinforcement learning uses a temporal 
difference method' [page 1, "reinforcement learning and function approximation, 
line 3-4]. 

11. As to claim 5, 10, 16, Sutton disclosed 'temporal difference method users on-line 
SARSA' [page 2, line 12-13, 25-27, fig 1], SARSA corresponds to sarsa algorithm as 
detailed in fig 1 ; 
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12. As to claim 6, 11, 17, Sutton disclosed 'temporal difference method using SARSA 
uses tile coding' [page 3, fig 1-2]. 

13. As to claim 7, Gall teaches a system that including 'memory management' [col 2, 
line 28-37, col 4, line 50-60, col 6, line 50-56]; 'a computer system [see fig 2, element 
30] or virtual machine [fig 3, element 44] having a memory space [fig 2, element 32a, 
col 4, line 50-60]; 'a memory space within said computer system [fig 2, element 
30,element 32a], computer system corresponds to fig 2, element 30, and memory 
corresponds to fig 2, element 32a; and 'accessible by the virtual machine for the runtime 
storage and execution of applications' [fig 2, col 8, line 11-15, line 25-33]; 'a garbage 
collector, allocation of memory to applications within said memory space' [col 8, line 33- 
35, COM3, line 1-8]. 

It is however, noted that Gall does not specifically teach 'reinforcement learning 
is used to control the management', although Gall specifically suggests virtual machine 
implementing Java Virtual Machine as detailed in fig 2, also teaches garbage collection 
related memory management algorithms [col 8, line 15-23]. On the other hand, Sutton 
disclosed 'reinforcement learning is used to control the management' see Abstract, 
page 1-2, specifically teaches "reinforcement learning applied for example in dynamic 
programming. 
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It would have been obvious to one of the ordinary skill in the art at the time of 
applicant's invention to incorporate the teachings of Sutton into relation-based ordering 
of objects in an object heap specifically directed to ordering, accessing data objects and 
their relationships between multiple objects in virtual memory scheme of Gall because 
both Sutton, and Gall directed to temporal and dynamic programming, more 
specifically, Gall is directed to heap managed garbage collection in multiple computer 
programs executing simultaneously on the same computer system [Gall: col 8, line 36- 
42, fig 3], while Sutton is directed to improving approximation, estimation based on 
reinforcement learning specifically implementing algorithms models that executes the 
task dynamics and applied dynamic programming backups [Sutton: page 2]. 

One of the ordinary skill in the art at the time of applicant's invention to 
incorporate the teachings of Sutton into relation-based ordering of objects in an object 
heap specifically directed to ordering, accessing data objects and their relationships 
between multiple objects in virtual memory scheme of Gall because that would have 
allowed users of Gall to use "reinforcement learning system" involving not only multiple 
step decision with respect to current actions, but also evaluates and actions under 
uncertainty or unprededictable actions because reinforcement learning algorithms 
supports state transitions improving overall performance as suggested [Sutton: page 2]. 
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Conclusion 
The prior art made of record 

a. US Patent No. 6480862 
b. Richard S Sutton, "Generalization in Reinforcement 
Learning: Successful Examples Using Sparse Coarse Coding", Advances in neural 
Information processing systems 8, 1996, pp 1038-1044. 



Application/Control Number: 10/630,525 



Art Unit: 2166 



Page 9 



Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Srirama Channavajjala whose telephone number is 
571-272-4108. The examiner can normally be reached on Monday-Friday from 
8:00 AM to 5:30 PM Eastern Time. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Alam, Hosain, T, can be reached on (571) 272-3978. The fax phone 
numbers for the organization where the application or proceeding is assigned is 
571-273-8300 Information regarding the status of an application may be obtained 
from the Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. 
Should you have questions on access to the Private PAIR system, contact the 
Electronic Business Center (EBC) at 866-217-9197 (toll-free) 



sc 

Patent Examiner. 
January 26, 2006. 




